Chained, scalable storage devices

ABSTRACT

Described embodiments access data in a chained, scalable storage system. A primary agent of one or more storage devices receives a host request including a logical address from a host coupled to the primary agent. The primary agent determines, based on the logical address, a corresponding physical address in at least one of the storage devices and generates, based on the physical address, a sub-request for each determined physical address in the storage devices. The primary agent sends, via a storage device interface network operable independently of the host, the sub-requests to the storage devices. The storage device interface network is a peer-to-peer network coupling the storage devices to the primary agent. The primary agent receives sub-statuses in response to the sub-requests, and determines an overall status. The primary agent provides the overall status to the host such that the host is coupled to the storage devices without a switch.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part, and claims the benefit ofthe filing date, of U.S. patent application Ser. No. 13/702,976, filedDec. 7, 2012, which claims the benefit of the filing date of U.S.provisional application No. 61/497,525 filed Jun. 16, 2011,International Patent Application no. PCT/US2011/040996 filed Jun. 17,2011, and U.S. provisional application No. 61/356,443 filed Jun. 18,2010, the teachings of all which are incorporated herein in theirentireties by reference.

BACKGROUND

A Storage Area Network (SAN) is a system that provides access toconsolidated, block-level storage, such as disk arrays and tapelibraries, to one or more host devices coupled to the SAN. A SANrepresents a plurality of storage devices as a single logical interfaceto the host devices, conceptually aggregating the storage implemented byeach of the storage devices into a single logical storage space. Atypical SAN might be scalable, meaning that the amount of storage space(e.g., the number of storage devices) can be changed as needed indifferent SAN systems. As noted, a SAN provides block-level access,meaning that the file system is typically managed by the host devices. Atypical SAN might employ block-level protocols such as Fibre Channel(FC), Advanced Technology Attachment (ATA) over Ethernet (AoE), InternetSmall Computer System Interface (iSCSI) or HyperSCSI. A SAN directlytransfers data between storage devices and host devices.

A Network Attached Storage (NAS) is a system that provides file-levelaccess to one or more host devices coupled to the NAS. Unlike a SAN, theNAS system provides a file system for its attached storage devices,essentially acting as a file server accessing one or more localblock-level storage devices. A typical NAS might employ file-levelprotocols such as Network File System (NFS) or Server MessageBlock/Common Internet File System (SMB/CIFS). A SAN-NAS hybrid system isa system that provides hosts with both file-level access like a NASdevice and block-level access like a SAN system from the same storagesystem.

In SAN, NAS and SAN-NAS hybrid systems, it is desired to employ multiplestorage devices such that the size of total system storage can beincreased by grouping together a plurality of storage devices. Suchgrouping of storage devices typically requires communication hierarchywith a switch such that the storage devices are available to the host,either individually or in aggregate.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Described embodiments access data in a chained, scalable storage system.A primary agent of one or more storage devices receives a host requestincluding a logical address from a host coupled to the primary agent.The primary agent determines, based on the logical address, acorresponding physical address in at least one of the storage devicesand generates, based on the physical address, a sub-request for eachdetermined physical address in the storage devices. The primary agentsends, via a storage device interface network operable independently ofthe host, the sub-requests to the storage devices. The storage deviceinterface network is a peer-to-peer network coupling the storage devicesto the primary agent. The primary agent receives sub-statuses inresponse to the sub-requests, and determines an overall status. Theprimary agent provides the overall status to the host such that the hostis coupled to the storage devices without a switch.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

Other aspects, features, and advantages of described embodiments willbecome more fully apparent from the following detailed description, theappended claims, and the accompanying drawings in which like referencenumerals identify similar or identical elements.

FIG. 1 shows a block diagram of a scalable storage system in accordancewith exemplary embodiments;

FIG. 2 shows a block diagram of a scalable storage system in accordancewith exemplary embodiments;

FIG. 3 shows a block diagram of a scalable storage system in accordancewith exemplary embodiments;

FIG. 4 shows a block diagram of a scalable storage system in accordancewith exemplary embodiments; and

FIG. 5 shows a block diagram of a scalable storage system in accordancewith exemplary embodiments.

DETAILED DESCRIPTION

Described embodiments access data in a chained, scalable storage system.A primary agent of one or more storage devices receives a host requestincluding a logical address from a host coupled to the primary agent.The primary agent determines, based on the logical address, acorresponding physical address in at least one of the storage devicesand generates, based on the physical address, a sub-request for eachdetermined physical address in the storage devices. The primary agentsends, via a storage device interface network operable independently ofthe host, the sub-requests to the storage devices. The storage deviceinterface network is a peer-to-peer network coupling the storage devicesto the primary agent. The primary agent receives sub-statuses inresponse to the sub-requests, and determines an overall status. Theprimary agent provides the overall status to the host such that the hostis coupled to the storage devices without a switch.

Table 1 defines a list of acronyms employed throughout thisspecification as an aid to understanding the described embodiments:

TABLE 1 AoE Advanced Technology CD Compact Disc Attachment (ATA) overEthernet DVD Digital Versatile Disc CIFS Common Internet File System HDDHard Disk Drive FC Fibre Channel IC Integrated Circuit HIF HostInterFace iSCSI Internet SCSI I/O Input/Output NAS Network AttachedStorage MRAM Magnetoresistive Random Access PCI-E Peripheral ComponentInterconnect Memory Express NFS Network File System RAID Redundant Arrayof Independent PHY PHysical Layer Disks RF Radio Frequency SAN StorageArea Network SAS Serial Attached SCSI SATA Serial Advanced TechnologySCSI Small Computer System Interface Attachment SoC System on Chip SMBServer Message Block SSD Solid-State Disk SRIO Serial Rapid Input/OutputUSB Universal Serial Bus

In some SAN, NAS or SAN-NAS hybrid systems, the storage devices mighthave a primary agent of the devices accept storage requests receivedfrom host devices over a host-interface (HIF) protocol. The primaryagent processes the host requests and generates one or more sub-requeststo secondary agents of each storage device over a peer-to-peer protocol.The secondary agents accept and process the sub-requests, and reportsub-status information for each of the sub-requests to the primary agentand/or the host. The primary agent optionally accumulates thesub-statuses into an overall status of the host request. Peer-to-peercommunication between the agents is optionally used to communicateredundancy information during host accesses and/or failure recoveries.Various failure recovery techniques might reallocate storage, reassignagents and recover data via redundancy information.

FIG. 1 shows a block diagram of an exemplary scalable storage system,for example as described in related U.S. patent application Ser. No.13/702,976, filed Dec. 7, 2012, which is incorporated herein byreference. As shown in FIG. 1, a scalable storage system includes atleast one host device (100) coupled to pluggable storage module 190 viacoupling 101. Coupling 101 might be implemented as a transmissionmedium, such as a backplane, copper cables, optical fibers, one or morecoaxial cables, one or more twisted pair copper wires, and/or one ormore radio frequency (RF) channels. For example, coupling 101 might beimplemented as an FC, AoE, iSCSI, or HyperSCSI link (e.g., in a SANsystem) or as an NFS or SMB/CIFS link (e.g., in a NAS system).

Pluggable storage module 190 includes at least one host/storage deviceinterface (shown as 180). Although shown in FIG. 1 as being integratedwith pluggable storage module 190, in some embodiments, host/storagedevice interface 180 might be integrated with each host device 100. Insome embodiments, pluggable storage module 190 might be implemented asan add-in card. As shown in FIG. 1, pluggable storage module 190includes host-visible storage 110, which includes one or more storagedevices 110(1)-110(N). Host-visible storage 110 implements storage, partor all of which is configured to allow access by host devices 100 viahost to storage device interface 180. Pluggable storage module 190 alsoincludes host-invisible storage 120, which includes one or more storagedevices 120(1)-120(M). Host-invisible storage 120 implements storagethat is not directly reported and, thus, “invisible,” to host devices100. However, the storage that is invisible to the host is reported andis indirectly accessible to host devices 100 by elements of host-visiblestorage 110, for example via a peer-to-peer protocol. For example, aprimary agent of the storage elements reports the combined storagecapacity of the primary agent and any secondary agents in communicationwith the primary agent, even though the secondary agents are not visibleto host device 100. In some embodiments, storage devices 110 and 120 arephysical storage devices, such as Solid State Disks (SSDs), Hard DiskDrives (HDDs), tape libraries, hybrid magnetic and solid state storagesystems, or some combination thereof.

Together, combinations of couplings 101, 111 and 121 enable request,status, and data transfers between host devices 100 and host-visiblestorage 110 (and host-invisible storage 120 via host-visible storage110). For example, one or more of the couplings enable transfers via ahost-interface protocol, for example by one of host devices 100operating as a master and one of the storage elements of host-visiblestorage 110 operating as a slave. Further, one or more of the couplingsenable transfers via a peer-to-peer protocol, for example by one of theelements of host-visible storage 110 operating as a primary agent andone of the elements of host-invisible storage 120 or another one of theelements of host-visible storage 110 operating as a secondary agent.Couplings 111 and 121 might be implemented as custom-designedcommunication links, or might be implemented as links conforming to astandard communication protocol such as, for example, a Small ComputerSystem Interface (SCSI) link, a Serial Attached SCSI (SAS) link, aSerial Advanced Technology Attachment (SATA) link, a Universal SerialBus (USB), a Fibre Channel (FC) link, an Ethernet link (e.g., a 10GElink), an IEEE 802.11 link, an IEEE 802.15 link, an IEEE 802.16 link, aPeripheral Component Interconnect Express (PCI-E) link, a Serial RapidI/O (SRIO) link, an InfiniBand link, or other similar interface link.

In some embodiments, host/storage device interface 180 might typicallybe implemented as one or more PCI-E or InfiniBand switches such thathost device 100, coupling 101 and host/storage device interface 180implement a unified switch. In further embodiments, the unified switchis operable as a transparent switch with respect to host-visible storage110 and also simultaneously operable as a non-transparent switch withrespect to host-invisible storage 120. As shown in FIG. 1, the PCI-Eswitch (e.g., host/storage device interface 180) is a separate elementdistinct from each of storage devices 110 and 120.

Thus, related U.S. patent application Ser. No. 13/702,976, filed Dec. 7,2012, incorporated herein by reference, describes a scalable storagesystem including one or more PCI-E or InfiniBand switches (e.g.,host/storage device interface 180). If the PCI-E switch is anon-transparent switch, details of the topology below the switch andspecifics of the configuration of individual storage devices is hiddenfrom the host device (e.g., on host initialization discovery of attacheddevices). Thus, employing the non-transparent switch, describedembodiments could select one of the storage devices to act as a masterdevice (e.g., a primary agent) to handle all host communication with allthe storage devices, and select the rest of the storage devices to actas slave devices (e.g., as a secondary agent) that are hidden from thehost device, even though all of storage devices 110 and 120 might beduplicate devices. Further, the aggregate group of storage devices mightappear as a single storage device to the host device.

Other described embodiments can provide scalable functionality withoutemploying a separate PCI-E switch by employing “neighbor-to-neighbor”communication such that communications employ point-to-point linksbetween each of the storage devices without a need for a higher level(e.g., a PCI-E hierarchy). By techniques such as routing or switching,all of the storage devices are able to communicate among each other eventhough all the connections are point-to-point between the storagedevices.

FIG. 2 shows a block diagram of an exemplary storage device 110. Hostdevice 100 is coupled to storage device 110 via coupling 101. Coupling100 is in communication with PHY interface 202. As shown in FIG. 2, PHYinterface 202 includes one or more upstream physical layer links orports (PHYs) (shown as 101) and one or more downstream PHYs (shown as218(1)-218(N)). As shown in FIG. 2, storage device 110 includes a massstorage device 216 that includes one or more of solid-state storage 210(e.g., an SSD), magnetic storage 212 (e.g., an HDD or tape library) andoptical storage 214 (e.g., a CD or DVD). Storage device 110 includesstorage interface 206, which communicates to each individual storagedevice 210, 212 and 214. Logical/Physical translation module 204translates between logical addresses for operations received from hostdevice 100 and physical addresses on mass storage 216. Storage device110 also includes sub-status module 222 and sub-request module 220, bothof which are in communication with PHY interface 202.

In described embodiments, the upstream PHYs (e.g., 101) are incommunication with a host device (e.g., 100) via the PCI-E hierarchy,and downstream PHYs (e.g., 218) are in communication with other storagedevices (e.g., multiple of 110). Exemplary embodiments might employ afixed number of configurable PHYs, for example, 8 total configurablePHYs, where a given PHY might be configured as an upstream link or adownstream link. Having configurable PHYs allows for a trade-off betweenbandwidth delivered to host 101 (e.g., upstream connectivity) andcapacity of the scalable storage system (e.g., downstream connectivity).Other embodiments might employ a fixed number of upstream PHYs and afixed number of downstream PHYs, for example, 2 upstream PHYs and 6downstream PHYs.

In various embodiments, some or all of PHYs 101 and 218 of a storagedevice (e.g., 110) might be operable at the same speed (e.g., a samemaximum speed) or might each be operable at different speeds. Forexample, some embodiments might allow each of PHYs 101 and 218 toindependently support any one or more of: PCI-E Gen1, Gen2, Gen3 orGen4, 10GE, InfiniBand, SAS, SATA, or a nonstandard protocol forcommunication with one or more storage devices. Each of PHYs 101 and 218are coupled to one or more respective PHY interfaces integrated withineach storage device 110. When, for example, PHY interface 202 is a PCI-Einterface, the PCI-E interface is configurable to communicate as one ormore of: a root complex; a forwarding point; and an endpoint. Aforwarding point is similar to a root complex in that a forwarding pointcan send and receive traffic among one or more PCI-E interfaces. A rootcomplex is additionally a root of a separate PCI-E hierarchy. Since ahost device (e.g., 100) coupled to one or more storage devices (e.g.,110) is itself a root complex, if one or more of the storage devicescoupled to the host also is a root complex, then a multi-root PCI-Ehierarchy is created.

Multiple of storage device 110 might be connected in any number ofdifferent ways. FIGS. 3-5 show block diagrams of exemplarypoint-to-point connections of multiple storage devices in scalablestorage systems in accordance with exemplary embodiments. As shown, invarious embodiments the PHYs and the PHY controllers might be coupledvia: a daisy chain (or optionally a loop) as shown in FIG. 3; a fixed,1-to-1 interconnection to a host device (shown in FIG. 4); a fullcrossbar topology; a partial crossbar topology; a multiplexor network; acombination thereof; or any other technique for coupling multiplehardware devices. In some embodiments, the connection network among thestorage devices is a switched network, while in others, the connectionnetwork among the storage devices is a routed network. Further, in someembodiments, at least some of storage devices 110 have a differentconfiguration of PHY, or one or more different types of PHYs (e.g.,PCI-E, 10GE, InfiniBand, SAS, SATA, etc.).

As shown in FIGS. 3 and 4, storage devices 110(A)-110(N) of FIG. 3, andstorage devices 110(1)-110(N) of FIG. 4 have internal PHY interfacesconfigured as forwarding points. FIG. 5 shows a hierarchical couplingwhere all of storage devices 110 have PHY interfaces that are configuredas forwarding points, except storage devices 110.Z1 through 110.ZN,which have PHY interfaces configured as endpoints. Thus, in describedembodiments, one or more of storage devices 110 (e.g., storage device110(A) of FIG. 3, 110(1)-110(N) of FIG. 4, 110.A of FIG. 5) is coupledto host device 100, and all of the storage devices are coupled directlyto host device 100 (e.g., as shown in FIG. 4), or are coupled indirectlyto host device 100 via others of the storage devices, without employing,for example, a PCI-E switch.

At least one of storage devices 110 acts as a primary agent, and atleast one or more of storage devices 110 act as secondary agents. Invarious embodiments, the one or more primary agents have a direct, moredirect, shorter, and/or lower latency connection with host device 100than the secondary agents. For example, as shown in FIG. 3, storagedevice 110(A) might act as the primary agent for storage devices110(B)-110(N), since, for example, storage device 110(A) has a directconnection to host device 100, while storage devices 110(B)-110(N) arecoupled to one another in a daisy chain. As shown in FIG. 4, all ofstorage devices 110(1)-110(N) are able to act as primary agents forthemselves, as each storage device 110(1)-110(N) has a direct connectionto host device 100. Each storage device having a direct connection tothe host advantageously enables bandwidth to/from the host to scalelinearly with a number of the storage devices. Further, having a subsetof the storage devices, such as just one of the storage devices, act asa primary agent and the others as secondary agents enables scalablecapacity without a need for the host to control a plurality of separatestorage devices. As shown in FIG. 5, storage device 110.A might act asthe primary agent for storage devices 110.B1-110.Bn, since, for example,storage device 110.A has a direct connection to host device 100, whilestorage device 110.B1 might act as a primary agent for storage devices(not shown) coupled via couplings 218(C1), and so on.

In described embodiments, all communication between primary agents andsecondary agents is performed as neighbor-to-neighbor traffic that isnot visible to host device 100 (and, thus, not visible to the PCI-Ehierarchy of host device 100). For example, as shown in FIG. 3, all ofthe neighbor-to-neighbor traffic is performed on couplings218(1)-218(N), and none of the neighbor-to-neighbor traffic is performedon connection 101 which couples storage devices 110 to host device 100.Similarly, as shown in FIG. 4, all of the neighbor-to-neighbor trafficis performed on couplings 218(1)-218(N), and none of theneighbor-to-neighbor traffic is performed on couplings 101(1)-101(N)coupling storage devices 110(1)-110(N) to host device 100. Similarly, asshown in FIG. 5, all of the neighbor-to-neighbor traffic is performed oncouplings 218(B1)-218(Zn), and none of the neighbor-to-neighbor trafficis performed on coupling 101 coupling storage device 110.A to hostdevice 100.

In described embodiments, the neighbor-to-neighbor traffic is controltraffic, such as the forwarding of commands received by a primary agentfrom host device 100 to a specific one of storage devices 110 andresponses (e.g., completions), back to a primary agent from the specificone of storage devices 110, information derived from commands receivedfrom host device 100, maintenance traffic such as synchronization orheartbeats; RAID or other data redundancy control or data traffic (e.g.,deltas for RAID), and other traffic. For example, when a write commandupdates a part of a RAID stripe on a particular one of storage devices110, the particular storage device sends a RAID delta to one or more ofthe other storage devices (e.g., the one of storage devices storing theRAID parity of the stripe) as neighbor-to-neighbor traffic.

Couplings 101 and 218, as shown in FIGS. 3-5, are optionally orselectively of different bandwidths and/or different protocols. Forexample, upstream connections (e.g., coupling 101) to host device 100might typically be PCI-E Gen4, while downstream connections (e.g.,couplings 218) among the various storage devices 110 might typically bePCI-E Gen3 or a different protocol, such as 10GE, InfiniBand, SAS, etc.Any of the couplings might have a different bandwidth or a differentnumber of physical links from each other. In some embodiments, controltraffic of any of couplings 101 and 218 might be transferred overrelatively lower-bandwidth sideband couplings, while data traffic mightbe transferred over relatively higher-bandwidth main band couplings.Thus, in some embodiments, any of couplings 101 and 218 might beimplemented as custom-designed communication links, or might beimplemented as links conforming to a standard communication protocolsuch as, for example, SCSI, SAS, SATA, USB, FC, Ethernet (e.g., 10GE),IEEE 802.11, IEEE 802.15, IEEE 802.16, PCI-E, SRIO, InfiniBand, or othersimilar interface link.

In some embodiments, such as shown in FIG. 4, a bandwidth upstream tohost device 100 is substantially equal to an aggregate deliverablebandwidth of the various storage devices 110. In some embodiments, suchas shown in FIG. 5, storage devices 110 that are communicatively closerto host device 100 (e.g., storage device 110.A) are configured for ahigher bandwidth than storage devices communicatively farther from hostdevice 100 (e.g., storage device 110.Z1). In some embodiments, each ofstorage devices 110 might have different capacities, capabilities, or beimplemented as different types of storage media, such as Solid StateDisks (SSDs), Hard Disk Drives (HDDs), Magnetoresistive Random AccessMemory (MRAM), tape libraries, hybrid magnetic and solid state storagesystems, or some combination thereof.

In some embodiments, a connection network among storage devices 110 usesa PCI-E protocol (or other standard protocol) but in nonstandard ways,such as by having a circular (loop) interconnection (e.g., as indicatedby optional coupling 218(N) in FIGS. 3 and 4). In further embodiments,the connection network among storage devices 110 is enabled to usenonstandard bandwidths, signaling, commands or protocol extensions toadvantageously improve performance. In general, the connection networkamong the storage devices 110 is enabled to provide inter-devicecommunication in a manner efficient in one or more of bandwidth,latency, and power.

Thus, as described herein, described embodiments access data in achained, scalable storage system. A primary agent of one or more storagedevices receives a host request including a logical address from a hostcoupled to the primary agent. The primary agent determines, based on thelogical address, a corresponding physical address in at least one of thestorage devices and generates, based on the physical address, asub-request for each determined physical address in the storage devices.The primary agent sends, via a storage device interface network operableindependently of the host, the sub-requests to the storage devices. Thestorage device interface network is a peer-to-peer network coupling thestorage devices to the primary agent. The primary agent receivessub-statuses in response to the sub-requests, and determines an overallstatus. The primary agent provides the overall status to the host suchthat the host is coupled to the storage devices without a switch.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment can be included in at least one embodiment. Theappearances of the phrase “in one embodiment” in various places in thespecification are not necessarily all referring to the same embodiment,nor are separate or alternative embodiments necessarily mutuallyexclusive of other embodiments. The same applies to the term“implementation.”

As used in this application, the word “exemplary” is used herein to meanserving as an example, instance, or illustration. Any aspect or designdescribed herein as “exemplary” is not necessarily to be construed aspreferred or advantageous over other aspects or designs. Rather, use ofthe word exemplary is intended to present concepts in a concretefashion.

While the exemplary embodiments have been described with respect toprocessing blocks in a software program, including possibleimplementation as a digital signal processor, micro-controller, orgeneral-purpose computer, described embodiments are not so limited. Aswould be apparent to one skilled in the art, various functions ofsoftware might also be implemented as processes of circuits. Suchcircuits might be employed in, for example, a single integrated circuit,a multi-chip module, a single card, or a multi-card circuit pack.

Described embodiments might also be embodied in the form of methods andapparatuses for practicing those methods. Described embodiments mightalso be embodied in the form of program code embodied in non-transitorytangible media, such as magnetic recording media, optical recordingmedia, solid state memory, floppy diskettes, CD-ROMs, hard drives, orany other non-transitory machine-readable storage medium, wherein, whenthe program code is loaded into and executed by a machine, such as acomputer, the machine becomes an apparatus for practicing describedembodiments. Described embodiments might can also be embodied in theform of program code, for example, whether stored in a non-transitorymachine-readable storage medium, loaded into and/or executed by amachine, or transmitted over some transmission medium or carrier, suchas over electrical wiring or cabling, through fiber optics, or viaelectromagnetic radiation, wherein, when the program code is loaded intoand executed by a machine, such as a computer, the machine becomes anapparatus for practicing the described embodiments. When implemented ona general-purpose processor, the program code segments combine with theprocessor to provide a unique device that operates analogously tospecific logic circuits. Described embodiments might also be embodied inthe form of a bitstream or other sequence of signal values electricallyor optically transmitted through a medium, stored magnetic-fieldvariations in a magnetic recording medium, etc., generated using amethod and/or an apparatus of the described embodiments.

It should be understood that the steps of the exemplary methods setforth herein are not necessarily required to be performed in the orderdescribed, and the order of the steps of such methods should beunderstood to be merely exemplary. Likewise, additional steps might beincluded in such methods, and certain steps might be omitted orcombined, in methods consistent with various described embodiments.

As used herein in reference to an element and a standard, the term“compatible” means that the element communicates with other elements ina manner wholly or partially specified by the standard, and would berecognized by other elements as sufficiently capable of communicatingwith the other elements in the manner specified by the standard. Thecompatible element does not need to operate internally in a mannerspecified by the standard. Unless explicitly stated otherwise, eachnumerical value and range should be interpreted as being approximate asif the word “about” or “approximately” preceded the value of the valueor range.

Also for purposes of this description, the terms “couple,” “coupling,”“coupled,” “connect,” “connecting,” or “connected” refer to any mannerknown in the art or later developed in which energy is allowed to betransferred between two or more elements, and the interposition of oneor more additional elements is contemplated, although not required.Conversely, the terms “directly coupled,” “directly connected,” etc.,imply the absence of such additional elements. Signals and correspondingnodes or ports might be referred to by the same name and areinterchangeable for purposes here.

It will be further understood that various changes in the details,materials, and arrangements of the parts that have been described andillustrated in order to explain the nature of the described embodimentsmight be made by those skilled in the art without departing from thescope expressed in the following claims.

We claim:
 1. A method of accessing data in a chained, scalable storagesystem, the method comprising: receiving, by a primary agent of one ormore storage devices, a host request from a host device coupled to theprimary agent via a host interface network, the request to access alogical address of the one or more storage devices; determining, by theprimary agent based on the logical address, a corresponding physicaladdress in at least one of the one or more storage devices; generating,by the primary agent based on the physical address, a sub-requestcorresponding to the host request and each of the determinedcorresponding physical addresses in at least one of the one or morestorage devices; sending, by the primary agent via a storage deviceinterface network operable independently of the host device, thesub-requests to the at least one storage device, the storage deviceinterface network a peer-to-peer network coupling the storage devices tothe primary agent; and receiving, by the primary agent from the at leastone storage device, respective sub-statuses in response to thesub-requests, determining an overall status based on each respectivesub-status, and providing the overall status to the host device, whereinthe host device is coupled to the one or more storage devices withoutemploying a network switch.
 2. The method of claim 1, wherein thestorage device interface network is not directly accessible to the hostinterface network.
 3. The method of claim 2, further comprising:sending, by each of the storage devices, data communication via arespective separate data communication path with the host separate fromthe storage device interface network, whereby control traffic betweenthe host device and the storage devices is solely between the hostdevice and the primary agent, while data communication bandwidth scaleswith a number of the storage devices.
 4. The method of claim 1, wherein,for the method, the host interface network and the storage deviceinterface network comprise transmission media comprising at least oneof: a backplane, one or more copper cables, one or more optical fibers,one or more coaxial cables, one or more twisted pair copper wires. 5.The method of claim 4, further comprising: selectively providing higherbandwidth storage device interface network connections to a subset ofthe one or more storage devices.
 6. The method of claim 5, wherein thesubset of the one or more storage devices comprises one or more of thestorage devices located proximately to the host device.
 7. The method ofclaim 4, wherein the host interface network comprises a PeripheralComponent Interconnect Express (PCI-E) network.
 8. The method of claim7, wherein the host interface network comprises a PCI-E Gen4 network,and the storage device interconnect network comprises one or more of: aPCI-E Gen3 network, an Ethernet network, a Serial Attached SmallComputer System Interface (SAS) network, and a Serial AdvancedTechnology Attachment (SATA) network.
 9. The method of claim 1, wherein,for the method, the one or more storage devices comprise at least oneof: a Solid State Disk (SSD), a Hard Disk Drive (HDD), aMagnetoresistive Random Access Memory (MRAM), a tape library, and ahybrid magnetic and solid state storage system.
 10. The method of claim1, further comprising: providing a bandwidth to the host interfacenetwork that is related to an aggregate deliverable bandwidth of the oneor more storage devices.
 11. The method of claim 10, wherein the storagedevice interface network comprises one or more physical links, each linkhaving an independent bandwidth.
 12. The method of claim 11, whereineach of the one or more physical links comprise (i) a relativelylower-bandwidth sideband coupling for transferring control data, and(ii) a relatively higher-bandwidth main band coupling for transferringuser data.
 13. The method of claim 10, wherein the providing comprisesproviding each of the storage devices with a separate physical link ofthe host interface network.
 14. The method of claim 1, furthercomprising: employing the one or more storage devices in a RedundantArray of Independent Disks (RAID) system.
 15. A chained, scalablestorage system comprising: a plurality of storage devices, at least oneof the storage devices a primary agent for one or more of the pluralityof storage devices; a host device coupled via a host interface networkto the at least one primary agent, wherein the at least one primaryagent is configured to: receive a host request from the host device, therequest to access a logical address of the one or more of the pluralityof storage devices; determine, based on the logical address, acorresponding physical address in at least one of the one or more of theplurality of storage devices; generate, based on the physical address, asub-request corresponding to the host request and each of the determinedcorresponding physical addresses in at least one of the one or more ofthe plurality of storage devices; send, via a storage device interfacenetwork operable independently of the host device, the sub-requests tothe at least one storage device, the storage device interface network apeer-to-peer network coupling the storage devices to the primary agent;and receive, from the at least one storage device, respectivesub-statuses in response to the sub-requests, determine an overallstatus based on each respective sub-status, and provide the overallstatus to the host device, wherein the host device is coupled to the oneor more storage devices without employing a network switch.
 16. Thesystem of claim 15, wherein the storage device interface network is notdirectly accessible to the host interface network.
 17. The system ofclaim 16, wherein control traffic between the host device and thestorage devices is solely between the host device and the at least oneprimary agent, and data bandwidth scales with a number of the storagedevices.
 18. The system of claim 15, wherein the storage deviceinterface network is configured to, at least one of: selectively providehigher bandwidth connections to a subset of the one or more storagedevices; and provide a bandwidth to the host interface network that isrelated to an aggregate deliverable bandwidth of the one or more storagedevices.
 19. The system of claim 15, wherein: the host interface networkcomprises a Peripheral Component Interconnect Express (PCI-E) Gen4network; the storage device interconnect network comprises one or moreof: a PCI-E Gen3 network, an Ethernet network, a Serial Attached SmallComputer System Interface (SAS) network, and a Serial AdvancedTechnology Attachment (SATA) network; and the one or more storagedevices comprise at least one of: a Solid State Disk (SSD), a Hard DiskDrive (HDD), a Magnetoresistive Random Access Memory (MRAM), a tapelibrary, a hybrid magnetic and solid state storage system; and aRedundant Array of Independent Disks (RAID).
 20. A non-transitorymachine-readable medium, having encoded thereon program code, wherein,when the program code is executed by a machine, the machine implements amethod of accessing data in a chained, scalable storage system, themethod comprising: receiving, by a primary agent of one or more storagedevices, a host request from a host device coupled to the primary agentvia a host interface network, the request to access a logical address ofthe one or more storage devices; determining, by the primary agent basedon the logical address, a corresponding physical address in at least oneof the one or more storage devices; generating, by the primary agentbased on the physical address, a sub-request corresponding to the hostrequest and each of the determined corresponding physical addresses inat least one of the one or more storage devices; sending, by the primaryagent via a storage device interface network operable independently ofthe host device, the sub-requests to the at least one storage device,the storage device interface network a peer-to-peer network coupling thestorage devices to the primary agent; and receiving, by the primaryagent from the at least one storage device, respective sub-statuses inresponse to the sub-requests, determining an overall status based oneach respective sub-status, and providing the overall status to the hostdevice, wherein the host device is coupled to the one or more storagedevices without employing a network switch.